Evaluating quality of a product such as a semiconductor substrate

ABSTRACT

An evaluation device may include: a receiving unit that receives an image of the semiconductor substrate, the image captured by an imaging device provided on the semiconductor substrate manufacturing apparatus; a determination unit that determines, using a neural network, at least one value representative of a probability of a machine learning device outputting an erroneous output for the image, the machine learning device configured to: (i) receive the image of the semiconductor substrate, (ii) perform computation using the received image, and (iii) output information indicating the quality of the semiconductor substrate based on a result of the computation; and an output unit that outputs an output based on the at least one value representative of the probability. The neural network has been trained using: images of manufactured semiconductor substrates; and information indicating, for each one of the images, a level of erroneousness for an output from the machine learning device.

The application relates to evaluating quality of a product such as asemiconductor substrate manufactured by a semiconductor substratemanufacturing apparatus.

BACKGROUND

Methods and/or systems for evaluating quality of manufactured productsusing artificial intelligence (AI) have been developed. For example, JP2008-164461A discloses an inspection method of a thin-plate shapedcomponent, where images of two surfaces of a thin-shaped component arecaptured, input values to a neural network are calculated from imagedata using two dimensional fast Fourier transform and the quality of thethin-plate shaped component is determined by inputting the calculatedinput values into the neural network.

However, determination made by AI might not always be correct. In thisrespect, JP H05-225163A discloses a neural network system that includesa problem solving neural network and an evaluation network, theevaluation network being trained to output an output node valuedepending on whether arbitrary input data values are learnt data usedfor training the problem solving neural network or are unlearnt datadifferent from the learnt data. The output node value from theevaluation network may be considered as indicating certainty of anoutput from the problem solving neural network for the arbitrary inputdata values.

In some circumstances, further improvements on more accurate evaluationon outputs of AI used for evaluating quality of manufactured productsare desirable in order to improve overall quality of manufacturedproducts.

SUMMARY

According to one aspect, an evaluation device is provided for evaluatingquality of a semiconductor substrate manufactured by a semiconductorsubstrate manufacturing apparatus. The evaluation device may comprisethe following:

-   -   a receiving unit configured to receive an image of the        semiconductor substrate, the image being captured by an imaging        device provided on the semiconductor substrate manufacturing        apparatus;    -   a determination unit configured to determine, using a neural        network, at least one value representative of a probability of a        machine learning device outputting an erroneous output for the        image of the semiconductor substrate, the machine learning        device being configured to: (i) receive the image of the        semiconductor substrate, (ii) perform computation using the        received image, and (iii) output information indicating the        quality of the semiconductor substrate based on a result of the        computation; and    -   an output unit configured to output an output based on the at        least one value representative of the probability,    -   wherein the neural network has been trained using:        -   images of manufactured semiconductor substrates; and        -   information indicating, for each one of the images of the            manufactured semiconductor substrates, a level of            erroneousness for an output from the machine learning device            for said one of the images of the manufactured semiconductor            substrates.

In some circumstances, the evaluation device according to variousaspects of the present disclosure may contribute to improving accuracyof evaluation of the quality of the semiconductor substrate by providingan output based on the at least one value representative of theprobability of the machine learning device outputting an erroneousoutput for the image of the semiconductor substrate. In other words, insome circumstances, the evaluation device according to various aspectsof the present disclosure may provide information indicating howreliable an output from the machine learning device is for a particularimage of a semiconductor substrate, which may lead to improved accuracyof evaluation of the quality of the semiconductor substrate.

The images used for training the neural network may include one or moreimages that are not included in training data used for training themachine learning device.

In some examples, the machine learning device may be configured tofurther receive sensor information from one or more sensors provided inrelation to the semiconductor substrate manufacturing apparatus and toperform the computation further using the sensor information. The one ormore sensors may be one or more of the following: a temperature sensor;a humidity sensor; a brightness sensor; an atmospheric pressure sensor.The neural network may have been trained further using the sensorinformation, and the determination made by the determination unit may bebased at least partially on the sensor information.

In some other examples, the neural network may have been trained furtherusing sensor information from one or more sensors provided in relationto the semiconductor substrate manufacturing apparatus. The one or moresensors may be one or more of the following: a temperature sensor; ahumidity sensor; a brightness sensor; an atmospheric pressure sensor.The determination made by the determination unit may be based at leastpartially on the sensor information.

Regarding the above stated aspect and examples, the evaluation devicemay further comprise:

-   -   a neural network training unit configured to train the neural        network using the images of the manufactured semiconductor        substrates and the information indicating a level of        erroneousness for an output from the machine learning device for        each of the images of the manufactured semiconductor substrates,    -   wherein the training of the neural network may be performed        according to deep learning technique, and    -   wherein the neural network training unit may be further        configured to generate the information used for training the        neural network by:        -   receiving the images of the manufactured semiconductor            substrates and quality information indicating, for each one            of the received images, quality of a manufactured            semiconductor substrate in said one of the received images;        -   providing one of the received images to the machine learning            device as an input;        -   obtaining an output from the machine learning device in            response to the provision of said one of the received            images; and        -   comparing the obtained output from the machine learning            device with the quality of the manufactured semiconductor            substrate in the image provided to the machine learning            device, the quality of the manufactured semiconductor            substrate being indicated in the received quality            information.

According to another aspect, an evaluation system is provided. Theevaluation system may comprise:

-   -   the evaluation device according to any one of the above stated        aspect and examples; and    -   the machine learning device configured to receive the image of        the semiconductor substrate, perform computation using the        received image, and output information indicating the quality of        the semiconductor substrate based on a result of the        computation.

In some examples, the evaluation system may further comprise:

-   -   an instruction generation unit configured to generate an        instruction to the semiconductor substrate manufacturing        apparatus as to processing of the semiconductor substrate based        on the output from the evaluation device and the output from the        machine learning device for the image of the semiconductor        substrate; and    -   a communication interface configured to communicate the        instruction to the semiconductor substrate manufacturing        apparatus.

According to yet another aspect, a semiconductor substrate manufacturingsystem is provided. The semiconductor substrate manufacturing system maycomprise:

-   -   the evaluation system according to any one of the above-stated        aspect and examples;    -   the semiconductor substrate manufacturing apparatus configured        to manufacture the semiconductor substrate; and    -   the imaging device provided on the semiconductor substrate        manufacturing apparatus,    -   wherein the semiconductor substrate manufacturing apparatus is        further configured to:        -   receive the instruction from the communication interface of            the evaluation system; and        -   process the semiconductor substrate according to the            received instruction.

According to yet another aspect, an evaluation method for evaluatingquality of a semiconductor substrate manufactured by a semiconductorsubstrate manufacturing apparatus is provided. The evaluation method maycomprise:

-   -   receiving, by a processor, an image of the semiconductor        substrate, the image being captured by an imaging device        provided on the semiconductor substrate manufacturing apparatus;    -   determining, by the processor, using a neural network, at least        one value representative of a probability of a machine learning        device outputting an erroneous output for the image of the        semiconductor substrate, the machine learning device being        configured to: (i) receive the image of the semiconductor        substrate, (ii) perform computation using the received        image, (iii) and output information indicating the quality of        the semiconductor substrate based on a result of the        computation; and    -   outputting, by the processor, an output based on the at least        one value representative of the probability,    -   wherein the neural network has been trained using:        -   images of manufactured semiconductor substrates; and        -   information indicating, for each one of the images of the            manufactured semiconductor substrates, a level of            erroneousness for an output from the machine learning device            for said one of the images of the manufactured semiconductor            substrates.

According to yet another aspect, a method is provided for training aneural network to determine at least one value representative of aprobability of a machine learning device outputting an erroneous outputfor an image of a semiconductor substrate, the machine learning devicebeing configured to: (i) receive the image of the semiconductorsubstrate, (ii) perform computation using the received image, and (iii)output information indicating quality of the semiconductor substratebased on a result of the computation. The method may comprise:

-   -   receiving images of manufactured semiconductor substrates and        information indicating, for each one of the images of the        manufactured semiconductor substrates, a level of erroneousness        for an output from the machine learning device for said one of        the images of the manufactured semiconductor substrates; and    -   training the neural network using the received images as inputs        to the neural network and the received information as        supervisory data, wherein the training may be according to deep        learning technique.

According to yet another aspect, a computer program product is provided.The computer program product may comprise computer-readable instructionsthat, when loaded and run on a computer, cause the computer to performthe method according to any one of the methods according to theabove-stated aspects.

According to yet another aspect, an evaluation device for evaluatingquality of a product manufactured by a manufacturing apparatus isprovided. The evaluation device may comprise:

-   -   a receiving unit configured to receive an image of the product,        the image being captured by an imaging device provided on the        manufacturing apparatus;    -   a determination unit configured to determine, using a neural        network, at least one value representative of a probability of a        machine learning device outputting an erroneous output for the        image of the product, the machine learning device being        configured to: (i) receive the image of the product, (ii)        perform computation using the received image, and (iii) output        information indicating the quality of the product based on a        result of the computation; and    -   an output unit configured to output an output based on the at        least one value representative of the probability,    -   wherein the neural network has been trained using:        -   images of manufactured products; and        -   information indicating, for each one of the images of the            manufactured products, a level of erroneousness for an            output from the machine learning device for said one of the            images of the manufactured products.

According to yet another aspect, an evaluation method for evaluatingquality of a product manufactured by a manufacturing apparatus isprovided. The evaluation method may comprise:

-   -   receiving, by a processor, an image of the product, the image        being captured by an imaging device provided on the        manufacturing apparatus;    -   determining, by the processor, using a neural network, at least        one value representative of a probability of a machine learning        device outputting an erroneous output for the image of the        product, the machine learning device being configured to: (i)        receive the image of the product, (ii) perform computation using        the received image, and (iii) output information indicating the        quality of the product based on a result of the computation; and    -   outputting, by the processor, an output based on the at least        one value representative of the probability,    -   wherein the neural network has been trained using:        -   images of manufactured products; and        -   information indicating, for each one of the images of the            manufactured products, a level of erroneousness for an            output from the machine learning device for said one of the            images of the manufactured products.

According to yet another aspect, a determination device is provided fordetermining error of an output from a machine learning device that isconfigured to: (i) receive input data in a predetermined format, (ii)perform computation using the input data, and (iii) provide a result ofthe computation as the output. The determination device may comprise:

-   -   a receiving unit configured to receive data having a format        corresponding to the predetermined format;    -   a determination unit configured to determine, using a neural        network, at least one value representative of a probability of        the machine learning device outputting an erroneous output for        input data corresponding to the received data; and    -   an output unit configured to output an output based on the at        least one value representative of the probability,    -   wherein the neural network has been trained using:        -   training data having the format corresponding to the            predetermined format; and        -   information indicating, for each element of the training            data, a level of erroneousness for an output from the            machine learning device for input data corresponding to the            element of the training data.

In various embodiments and examples described herein, the “predeterminedformat” of the input data for the machine learning device may be aformat suitable for computation performed by the machine learningdevice. For example, in case the machine learning device is configuredto process image data, the “predetermine format” may represent a formatof image data including pixels with intensity values. Other examples ofthe “predetermined format” of the input data for the machine learningdevice are described later herein.

In various embodiments and examples described herein, the term “a formatcorresponding to the predetermined format” may be understood asincluding a format identical to the predetermined format. Further, invarious embodiments and examples described herein, a “format” of datamay be understood as information specifying parameters to be included inthe data having that format. Thus, in various embodiments and examplesdescribed herein, “data having a format corresponding to thepredetermined format” may include at least one parameter that isidentical to a parameter included in data having the predeterminedformat.

According to yet another aspect, a computer-implemented method isprovided for determining error of an output from a machine learningdevice that is configured to: (i) receive input data in a predeterminedformat, (ii) perform computation using the input data, and (iii) providea result of the computation as the output. The method may comprise:

-   -   receiving data having a format corresponding to the        predetermined format;    -   determining, using a neural network, at least one value        representative of a probability of the machine learning device        outputting an erroneous output for input data corresponding to        the received data; and    -   outputting an output based on the at least one value        representative of the probability,    -   wherein the neural network has been trained using:        -   training data having the format corresponding to the            predetermined format; and        -   information indicating, for each element of the training            data, a level of erroneousness for an output from the            machine learning device for input data corresponding to the            element of the training data.

The subject matter described in the application can be implemented as amethod or as a system, possibly in the form of one or more computerprogram products. The subject matter described in the application can beimplemented in a data signal or on a machine readable medium, where themedium is embodied in one or more information carriers, such as aCD-ROM, a DVD-ROM, a semiconductor memory, or a hard disk. Such computerprogram products may cause a data processing apparatus to perform one ormore operations described in the application.

In addition, subject matter described in the application can also beimplemented as a system including a processor, and a memory coupled tothe processor. The memory may encode one or more programs to cause theprocessor to perform one or more of the methods described in theapplication. Further subject matter described in the application can beimplemented using various machines.

BRIEF DESCRIPTION OF THE DRAWINGS

Details of one or more implementations are set forth in the exemplarydrawings and description below. Other features will be apparent from thedescription, the drawings, and from the claims. It should be understood,however, that even though embodiments are separately described, singlefeatures of different embodiments may be combined to furtherembodiments.

FIG. 1 shows an exemplary hardware configuration of an evaluation systemaccording to an exemplary embodiment.

FIG. 2 shows an exemplary functional block diagram of the evaluationsystem.

FIG. 3 shows an exemplary functional block diagram of a machine learningdevice.

FIG. 4 shows a schematic diagram illustrating an exemplary input layerand an exemplary convolutional layer of a convolutional neural network(CNN).

FIG. 5 shows a schematic diagram illustrating an exemplary max poolingoperation.

FIG. 6 shows an exemplary configuration of a CNN.

FIG. 7A shows an exemplary configuration of a neural network which maybe trained using a deep learning technique.

FIG. 7B shows how a hidden layer of the neural network shown in FIG. 7Acan be trained in some examples.

FIG. 8 shows a flowchart of exemplary processing performed by theevaluation system.

FIG. 9 shows a flowchart of exemplary processing for training a machinelearning device (parent AI).

FIG. 10 shows a flowchart of exemplary processing for training a neuralnetwork of an evaluation device comprised in the evaluation system.

FIG. 11 shows a flowchart of exemplary processing performed by themachine learning device (parent AI).

FIG. 12 shows a flowchart of exemplary processing performed by theevaluation device.

FIG. 13 shows a flowchart of exemplary processing for generating aninstruction to a manufacturing apparatus.

DETAILED DESCRIPTION OF EMBODIMENTS

In the following text, a detailed description of examples will be givenwith reference to the drawings. It should be understood that variousmodifications to the examples may be made. In particular, elements ofone example may be combined and used in other examples to form newexamples.

Hardware Configuration

FIG. 1 shows an exemplary hardware configuration of an evaluation systemaccording to an exemplary embodiment. In FIG. 1, a system 1 comprises anevaluation device 10, a manufacturing apparatus 40, a camera 50 and oneor more sensors 60-1, . . . , 60-N.

The evaluation device 10 may be implemented by a general purposecomputer. For example, as shown in FIG. 1, the evaluation device 10 maycomprise a processor 12, a system memory 14, hard disk drive (HDD)interface 16, external disk drive interface 20, and input/output (I/O)interfaces 24. These components of the evaluation device 10 are coupledto each other via a system bus 30. The processor 12 may performarithmetic, logic and/or control operations by accessing the systemmemory 14. The system memory 14 may store information and/orinstructions for use in combination with the processor 12. The systemmemory 14 may include volatile and non-volatile memory, such as a randomaccess memory (RAM) 140 and a read only memory (ROM) 142. A basicinput/output system (BIOS) containing the basic routines that helps totransfer information between elements within the general purposecomputer, such as during start-up, may be stored in the ROM 142. Thesystem bus 30 may be any of several types of bus structures including amemory bus or memory controller, a peripheral bus, and a local bus usingany of a variety of bus architectures.

The evaluation device shown in FIG. 1 may include a hard disk drive(HDD) 18 for reading from and writing to a hard disk (not shown), and anexternal disk drive 22 for reading from or writing to a removable disk(not shown). The removable disk may be a magnetic disk for a magneticdisk drive or an optical disk such as a CD ROM for an optical diskdrive. The HDD 18 and the external disk drive 22 are connected to thesystem bus 30 by a HDD interface 16 and an external disk drive interface20, respectively. The drives and their associated computer-readablemedia provide non-volatile storage of computer-readable instructions,data structures, program modules and other data for the general purposecomputer. The data structures may include relevant data for theimplementation of one or more methods according to various aspects andexamples as described herein. The relevant data may be organized in adatabase, for example a relational or object database.

Although the exemplary environment described herein employs a hard disk(not shown) and an external disk (not shown), it should be appreciatedby those skilled in the art that other types of computer readable mediawhich can store data that is accessible by a computer, such as magneticcassettes, flash memory cards, digital video disks, random accessmemories, read only memories, and the like, may also be used in theexemplary operating environment.

A number of program modules may be stored on the hard disk, externaldisk, ROM 142 or RAM 140, including an operating system (not shown), oneor more application programs 1402, other program modules (not shown),and program data 1404. The application programs and correspondingmethods may include at least a part of the functionality as will bedescribed below, referring to FIGS. 2 to 13.

The evaluation device 10 shown in FIG. 1 may also include an inputdevice 26 such as mouse and/or keyboard, and display device 28, such asliquid crystal display. The input device 26 and the display device 28are connected to the system bus 30 via I/O interfaces 24 d, 24 e.

It should be noted that the above-described evaluation device 10employing a general purpose computer is only one example of animplementation of the exemplary embodiments described herein. Forexample, the evaluation device 10 may include additional components notshown in FIG. 1, such as one or more network interfaces forcommunicating with other devices and/or computers via wired and/orwireless communication for data exchange.

In addition or as an alternative to an implementation using a generalpurpose computer as shown in FIG. 1, a part or all of the functionalityof the exemplary embodiments described herein may be implemented as oneor more hardware circuits. Examples of such hardware circuits mayinclude but are not limited to: Large Scale Integration (LSI),Application Specific Integrated Circuit (ASIC) and Field ProgrammableGate Array (FPGA).

The camera 50 shown in FIG. 1 may be an imaging device comprising, e.g.,a CCD sensor that can capture one or more images of a scene. The camera50 may be connected to the system bus 30 of the general purpose computerimplementing the evaluation device 10 via the I/O interface 24 a inwired and/or wireless communication. In various exemplary embodiments asdescribed herein, the camera 50 may capture one or more images of anobject or a part thereof that is subject to evaluation and/orclassification. The object may be a product manufactured by amanufacturing apparatus. Examples of the product may include, but maynot be limited to, a semiconductor substrate, a substrate with solderedcomponents, a resin substrate and a liquid crystal. For example, thecamera 50 shown in FIG. 1 may capture an image of a semiconductorsubstrate manufactured by a manufacturing apparatus 40. Thesemiconductor substrate may be a substrate for an integrated circuit(IC), for example. An image captured by the camera 50 may include a 2Darray of pixels. Each of the pixels may include at least one value. Forexample, a pixel in a grey scale image may include one value indicatingan intensity of the pixel. A pixel in a color image may include multiplevalues, for example three values, that indicate coordinates in a colorspace such as RGB color space.

The manufacturing apparatus 40 may be an apparatus configured tomanufacture products. In some examples, the manufacturing apparatus 40may be a semiconductor substrate manufacturing apparatus configured tomanufacture semiconductor substrates. The camera 50 may be provided onthe manufacturing apparatus 40, at a location suitable for capturing animage of a product manufactured by the manufacturing apparatus 40. Themanufacturing apparatus 40 may be connected to the system bus 30 of thegeneral purpose computer implementing the evaluation device 10 via theI/O interface 24 b.

Further, in some examples, one or more sensors 60-1, . . . , 60-N may beconnected to the system bus 30 of the general purpose computerimplementing the evaluation device 10 via the I/O interface 24 c inwired and/or wireless communication. Each of the sensors 60-1, . . . ,60-N may be configured to detect a value of a physical parameter thatmay represent an environment in which the manufacturing apparatus 40 isprovided. For example, the one or more sensors 60-1, . . . , 60-N may beone or more of: a temperature sensor, a humidity sensor, a brightnesssensor, an atmospheric pressure sensor. In other examples where sensorvalues are not required for the processing performed by the evaluationdevice 10, no sensor may be connected to the system bus 30 of thegeneral purpose computer implementing the evaluation device 10 via theI/O interface 24 c.

In the following, exemplary embodiments will be described with respectto an exemplary case where the manufacturing apparatus 40 is asemiconductor substrate manufacturing apparatus and quality ofsemiconductor substrates manufactured by the manufacturing apparatus 40are to be evaluated. It should be noted, however, other examples of thepresent disclosure may be employed for applications other thanevaluating the quality of manufactured semiconductor substrates, as willbe described later herein.

Functional Configurations

FIG. 2 shows an exemplary functional block diagram of the evaluationsystem. The evaluation system shown in FIG. 2 may include the evaluationdevice 10, the camera 50, a machine learning device 70, a neural networkdatabase (DB) 80 and/or an instruction generation unit 90.

The machine learning device 70 may be a device for evaluatingmanufactured semiconductor substrates using AI. For example, the machinelearning device 70 may evaluate quality of semiconductor substrates andclassify the semiconductor substrates into a specific (predetermined orpredeterminable) number of groups according to the quality. An outputfrom the machine learning device 70 may be evaluated by the evaluationdevice 10. Hereinafter, the machine learning device 70 is referred toalso as a parent AI.

In order for evaluating products such as manufactured semiconductorsubstrates, the machine learning device 70 may be configured to: (i)receive an image of a semiconductor substrate from the camera 50provided on the manufacturing apparatus 40, (ii) perform computationusing the received image, and (iii) output information indicating thequality of the semiconductor substrate in the received image based onthe result of the computation. The computation may be based on a knownmachine learning technique, for example, a technique involving neuralnetworks.

FIG. 3 shows an exemplary functional block diagram of the machinelearning device 70. The machine learning device 70 shown in FIG. 3 mayinclude a receiving unit 700, a computation unit 702, an output unit704, a training data generation unit 706 and/or an AI training unit 708.

The receiving unit 700 may be configured to receive inputs to themachine learning device 70 from other device(s). For example, as shownin FIG. 3, the receiving unit 700 may be configured to receive an imageof a semiconductor substrate from the camera 50 provided on themanufacturing apparatus 40. Further, for example, the receiving unit 700may be configured to receive sensor information from one or more sensors60 (not shown in FIG. 3), if necessary for the computation made by themachine learning device 70. The receiving unit 700 may be furtherconfigured to receive user inputs from an input device (not shown) suchas a mouse or a keyboard.

The computation unit 702 may be configured to perform computation usingthe input(s) received by the receiving unit 700. For instance, thecomputation unit 702 may be configured to perform computation using theimage received from the camera 50. The computation unit 702 may includean AI 7020 which is used for the computation. In some examples, the AI7020 may comprise a convolutional neural network (CNN) that is known asa neural network suitable for image recognition. An exemplaryapplication of a CNN to evaluation of a semiconductor substrate will bedescribed below with reference to FIGS. 4 to 6.

FIG. 4 shows a schematic diagram illustrating an exemplary input layerand an exemplary convolutional layer of a CNN. In the CNN shown in FIG.4, an input image having W×W (W=1, 2, 3, . . . ) pixels for K (K=1, 2,3, . . . ) channels (e.g., three channels corresponding to Red, Greenand Blue) can be input to the input layer. In this example, the inputimage may be an image of a semiconductor substrate. An intensity valueof a pixel for a channel can be considered as an input value to an inputnode of the input layer. In other words, the input layer may includeW×W×K input nodes, each of which corresponding to an intensity value ofa channel of a pixel.

Each node of the convolutional layer of the CNN shown in FIG. 4 maycorrespond to a filter having a size of F×F (F=1, 2, 3, . . . ; F<W),applied to a part of the input image. As shown in FIG. 4, M (M=1, 2, 3,. . . ) filters may be applied to the same part of the input image overthe K channels. An output of each node in the convolutional layer may berepresented as follows by equation (1):

y=f(Σ_(i=0) ^(F×F×K-1) w _(i) x _(i) +b)  (1)

where x_(i) may represent an input value to an input node (e.g., anintensity value of a pixel for a channel within the region covered bythe corresponding filter); w_(i) may represent an adjustable weight fora connection between the node in the convolutional layer and the inputnode corresponding to x_(i); and b may represent a bias parameter. Theactivation function f may be a rectified linear unit, f(x)=max(x, 0).

In some examples, each of the M filters may be applied to the whole areaof the input image by sliding the filter with a stride of S pixel(s) inboth width and height directions shown in FIG. 4. For each location ofthe M filters on the input image, M nodes corresponding to the M filtersmay be present in the convolutional layer. In case of S=1, the number ofoutputs of the convolutional layer may be W×W×M. The outputs of theconvolutional layer may be considered as M images (corresponding to Mfilters) with a size of W×W.

The outputs of the convolutional layer may be subject to down-samplingby a max pooling operation. The max pooling operation may select themaximum value among a plurality of input values. The max poolingoperation may be applied to each of the M images with a size of W×W,output from the convolutional layer as stated above.

FIG. 5 shows a schematic diagram illustrating an exemplary max poolingoperation. In the exemplary max pooling operation as shown in FIG. 5,filters having a size of 2×2 may be applied to an input image (to themax pooling operation) with a stride of two pixels. This may result inan output image including pixels each of which has the maximum intensityvalue among the pixels of the input image within the correspondingfilter. Each filter used in the max pooling operation may be consideredas a node of a pooling layer comprised in a CNN.

The outputs of the pooling layer may be input to another convolutionallayer. Alternatively, the outputs of the pooling layer may be input to aneural network called fully connected neural network, where each node ofthe fully connected neural network is connected to all the outputs (e.g.nodes) of the pooling layer. The outputs of the fully connected neuralnetwork may be connected either to another fully connected neuralnetwork or an output layer.

The output layer may include one or more nodes corresponding to one ormore desired output parameters of the CNN. For example, in the exemplaryembodiments, the output layer may include an output node indicating thequality of the semiconductor substrate in the input image captured bythe camera 50. In other examples, the output layer may include twooutput nodes, one indicating a likelihood that the quality of thesemiconductor substrate in the input image is acceptable for itsintended purpose, the other indicating a likelihood that the quality isnot acceptable. In yet other examples, the output layer may includethree or more output nodes, each of which indicating a likelihood thatthe quality of the semiconductor substrate in the input image belongs toone of the predefined classes of quality (e.g., high, ordinary, lowetc.). Each output node may comprise a softmax function as theactivation function. When the output layer includes two or more nodes,the CNN may be considered as solving a classification problem toclassify the semiconductor substrate in the input image into one of aspecified (predetermined or predeterminable) number of groups.

FIG. 6 shows an exemplary configuration of a CNN. The CNN as shown inFIG. 6 includes an input layer, a convolutional layer 1, a pooling layer1, a convolutional layer 2, a pooling layer 2, a fully connected layerand an output layer. The convolutional layers 1, 2 and the poolinglayers 1, 2 may have the configurations as explained above withreference to FIGS. 4 and 5. As also mentioned above, a CNN may includemore pairs of a convolutional layer and a pooling layer. Further, a CNNmay include a sequence of convolutional layers without having a poolinglayer in between the adjacent convolutional layers, as long as the lastconvolutional layer of the sequence is connected to a pooling layer.Further, a CNN may include more than one fully connected layers rightbefore the output layer.

Further details of known CNN techniques which may be applied inconnection with the present disclosure may be found in, for example,Okatani, “Deep Learning and Image Recognition,—Basics and CurrentTrends—” (in the Japanese language), Operations research as a managementscience research, 60(4), p. 198-204, The Operations Research Society ofJapan, Apr. 1, 2015, and Anonymus, “Convolutional neural network”,Wikipedia (URL:https://en.wikipedia.org/wiki/Convolutional_neural_network).

Referring again to FIG. 3, the output unit 704 of the machine learningdevice 70 may be configured to output information indicating quality ofthe semiconductor substrate in the received image based on a result ofthe computation performed by the computation unit 702. For instance, theinformation output from the output unit 704 may indicate whether or notthe quality of the semiconductor substrate is acceptable to be used forthe intended purpose of the semiconductor substrate. In some exampleswhere the AI 7020 of the computation unit 702 comprises a CNN asdescribed above, the output unit 704 may be configured to receive theoutput value(s) of the output node(s) of the CNN and to outputinformation indicating the quality of the semiconductor substrate basedon the received output value(s). For example, in case the CNN has twooutput nodes, one indicating a likelihood of acceptable quality and theother indicating a likelihood of inacceptable quality, the output unit704 may output information indicating that the quality is acceptable orinacceptable, depending on which output node has output a higher value.The output from the output unit 704 may be provided to the evaluationdevice 10 and/or the manufacturing apparatus 40, for example.

The training data generation unit 706 may be configured to generatetraining data for training the AI 7020 of the computation unit 702. Forinstance, in case the AI 7020 comprises a CNN as stated above, thetraining data may include a set of combinations of an input image, e.g.,of a semiconductor substrate, and a desired output for the input image,e.g., information indicating quality of the semiconductor substrate inthe image. The information indicating quality of the semiconductorsubstrate may be obtained from a user input, for instance. Further, forexample, the set of images and the quality information may be obtainedby capturing images of semiconductor substrates that are known to have acertain quality. The desired output for the input image may be setaccording to the number of output nodes included in the output layer ofthe CNN. The desired output may be considered as a supervisory signalfor training the AI 7020.

The AI training unit 708 may be configured to train the AI 7020 usingthe training data generated by the training data generation unit 706.For instance, when the AI 7020 comprises a CNN as stated above, the AItraining unit 708 may be configured to adjust the adjustable weights ofthe convolutional layer(s) included in the CNN (see equation (1) above)and the weights of the fully connected neural network(s) included in theCNN (see e.g., FIG. 6) by backpropagation method using the trainingdata.

In some examples, the AI training unit 708 may be configured to obtain adata structure of the CNN from the neural network DB 80.

The neural network DB 80 may be a database storing data structures ofneural networks with various configurations. For example, the neuralnetwork DB 80 may store the data structures of neural networks having aninput layer with various numbers of nodes, one or more hidden layerswith various numbers of nodes, an output layer with various numbers ofnodes and various weighted connections between nodes. Further, forexample, the neural network DB 80 may store the data structures of CNNas explained above with reference to FIGS. 4 to 6. The neural networksstored in the neural network DB 80 may not have been trained for anyspecific purpose.

In some examples, the training data generation unit 706 and/or the AItraining unit 708 may be provided in a device separate from the machinelearning device 70 rather than as a part of the machine learning device70.

The machine learning device 70 as described above with reference to FIG.3 may be implemented using a general purpose computer having aconfiguration analogous to that of the general purpose computer shown inFIG. 1 for implementing the evaluation device 10.

Referring again to FIG. 2, the evaluation device 10 may include areceiving unit 100, a determination unit 102, an output unit 104 and aneural network training unit 106.

The receiving unit 100 may be configured to receive an image of asemiconductor substrate, captured by the camera 50 provided on themanufacturing apparatus 40. Further, the receiving unit 100 may beconfigured to receive an output from the machine learning device 70, theparent AI. Further, for example, the receiving unit 100 may beconfigured to receive sensor information from one or more sensors 60(not shown in FIG. 2), if necessary for the determination made by thedetermination unit 102. The receiving unit 100 may be further configuredto receive user inputs from an input device (not shown) such as a mouseor a keyboard.

The determination unit 102 may include a neural network (NN) 1020. Thedetermination unit 102 may be configured to determine, using the NN1020, at least one value representative of a probability of the parentAI outputting an erroneous output for the image of the semiconductorsubstrate. The NN 1020 may be trained by the neural network trainingunit 106.

The neural network training unit 106 may be configured to train the NN1020 for determining at least one value representative of a probabilityof the parent AI outputting an erroneous output for specified inputdata. The data structure of the NN 1020 may be obtained from the neuralnetwork DB 80. In the exemplary embodiments, the NN 1020 may be trainedto output a value indicating or representative of a probability of theparent AI outputting an erroneous output for a particular image of asemiconductor substrate. In some examples, the neural network trainingunit 106 may be configured to train the neural network using trainingdata including: images of manufactured semiconductor substrates; andinformation indicating, for each one of the images of the manufacturedsemiconductor substrates, a level of erroneousness for an output fromthe parent AI for said one of the images of the manufacturedsemiconductor substrates. The information indicating the level oferroneousness may represent, for example, whether or not the parent AIhas output an erroneous output for said one of the images of themanufactured semiconductor substrates.

The information indicating the above-stated level of erroneousness maybe obtained by, for example, inputting into the parent AI the particularimage for which the quality of the semiconductor substrate in the imageis known, obtaining an output from the parent AI and comparing theoutput from the parent AI with the known quality of the semiconductorsubstrate in the image. The information indicating the above-statedlevel of erroneousness may be used as a supervisory signal for trainingthe NN 1020.

It should be noted that the set of images included in the training datafor training the NN 1020 and the set of images included in the trainingdata for training the AI 7020 of the parent AI may be identical, or havea partial overlap, or have no overlap. In preferred examples, the set ofimages included in the training data for training the NN 1020 maycomprise at least one image that is not comprised in the set of imagesincluded in the training data for training the AI 7020 of the parent AI.

Although FIG. 2 shows the neural network training unit 106 as a part ofthe evaluation device 10, in some examples, the neural network trainingunit 106 may be provided on a device different from the evaluationdevice 10. In such examples, the evaluation device 10 may obtain, fromthe neural network training unit 106, the NN 1020 that has been trainedby the neural network training unit 106.

In some examples, the NN 1020 may have the same configuration as the CNNas described above with reference to FIGS. 4 to 6. The output of the CNNcomprised in the determination unit 102, however, may indicate theprobability of the parent AI outputting an erroneous output, rather thanthe quality of the semiconductor substrate in the image. The trainingfor the CNN as the NN 1020 to output the probability of the parent AIoutputting an erroneous output may be performed in a manner analogous tothat stated above for training the CNN as the AI 7020 of the parent AI,but using the information indicating a level of erroneousness for anoutput from the parent AI for said one of the images of the manufacturedsemiconductor substrates as the supervisory signal, instead of theinformation indicating the quality of the semiconductor substrates inthe input images.

In other examples, the NN 1020 may have a configuration as shown in FIG.7A and may be trained using a known deep learning technique involving anautoencoder.

A neural network to be trained by a known deep learning technique maycomprise more than three layers in total, including an input layer(e.g., layer L0 in FIG. 7A), two or more hidden layers (e.g., layers L1,L2 in FIG. 7A) and an output layer (e.g., layer L3 in FIG. 7A). AlthoughFIG. 7A shows four layers, the neural network for deep learning may havemore than four layers, e.g. more than two hidden layers. Further, eachlayer in the neural network for deep learning may have more number orless number of nodes than that shown in FIG. 7A. For example, in theexemplary embodiments where an image of a semiconductor substrate isprocessed, the input layer may include input nodes each of whichincludes an intensity value of a pixel in an input image of asemiconductor substrate.

When training the neural network as shown in FIG. 7A, weights ofconnections to each hidden layer of the neural network may be adjustedso as to build an autoencoder that learns a representation (e.g.,encoding) for a set of data. For example, in order to train the hiddenlayer L2 shown in FIG. 7A, an autoencoder having a neural network shownin FIG. 7B may be constructed and trained. Referring to FIG. 7B, thelayer L1 may be considered as an input layer connected to the hiddenlayer L2 and an output layer having the same number of nodes as theinput layer L1 may be provided. It is noted that the layers L1 and L2 inFIG. 7B correspond to the layers L1 and L2 in FIG. 7A. The autoencodershown in FIG. 7B may be trained using the input data to the input layeras the supervisory signal. In other words, the weights of theconnections to the hidden layer L2 may be adjusted so that the outputlayer outputs the same data as the input data. Performing such trainingmay result in the hidden layer of the autoencoder to representcompressed information of the input data, in other words, representcharacteristics or features of the input data. The training of anautoencoder as shown in FIG. 7B may be iterated for each of the hiddenlayers of the neural network as shown in FIG. 7A.

Several techniques may be applied for improving robustness of anautoencoder. For example, partially corrupted input (e.g., input withadded noise) may be used while training the autoencoder to recover theoriginal undistorted input. Further, for example, sparsity may beimposed on the hidden layer (e.g., providing more nodes in the hiddenlayer than in the input layer) during training and the autoencoder maybe trained so that only a specified percentage of nodes in the hiddenlayer are active. For further example, one or more nodes in the hiddenlayer may be made inactive during training.

Either in case of employing the CNN (see e.g., FIGS. 4 to 6) or theneural network trained using an autoencoder (see FIGS. 7A and 7B) as theNN 1020 of the evaluation device 10, the output layer of the NN 1020 mayinclude, for instance, a single node that may output one or more valuesrepresentative of the probability of the parent AI outputting anerroneous output for the input data. In case of using a single node inthe output layer, the desired output value (e.g. supervisory signal) ofthe single output node may be set to a predetermined value (e.g., 0)when the parent AI has output a correct output for the input data and toanother predetermined value (e.g., 1) when the parent AI has output anerroneous output for the input data. Further, for example, the outputlayer of the NN 1020 may include two nodes, one corresponding to aprobability of the parent AI outputting a correct output for the inputdata, the other corresponding to a probability of the parent AIoutputting an erroneous output for the input data.

Referring again to FIG. 2, the output unit 104 may be configured tooutput an output based on the at least one value representative of theprobability determined by the determination unit 102. In some examples,the output by the output unit 104 may be identical to the output by theNN 1020 of the determination unit 102. In other examples, an outputvalue output from the output unit 104 may be determined based on theoutput by the NN 1020 of the determination unit 102. For instance, incase the NN 1020 outputs a single value y (e.g., 0≤y≤1.0) indicating orrepresentative of the probability of the parent AI outputting anerroneous output, the output value from the output unit 104 mayrepresent a percentage value corresponding to the probability.Alternatively, the output value from the output unit 104 may take eitherone of two values, one corresponding to correct and the othercorresponding to erroneous, determined by, for example, checking whetheror not the single output value y of the NN 1020 exceeds a predeterminedthreshold. Further, also in case the NN 1020 outputs two values, eachindicating or being representative of the probability of the parent AIoutputting an erroneous or correct output, the output value from theoutput unit 106 may represent a percentage value corresponding to theprobability of the parent AI outputting an erroneous (or correct) outputor may take either one of two values, one corresponding to correct andthe other corresponding erroneous, determined by, for example, selectingthe more probable of the two outputs from the NN 1020.

The output from the output unit 106 may be provided to the instructiongeneration unit 90.

The instruction generation unit 90 may be configured to generate aninstruction to a manufacturing apparatus 40 as to processing of thesemiconductor substrate based on the output from the output unit 106 ofthe evaluation device 10 and the output from the parent AI for the imageof the semiconductor substrate. For example, in case the parent AI isconfigured to output whether or not the quality of the semiconductorsubstrate is acceptable, the following combinations of the outputs fromthe parent AI and the output unit 106 of the evaluation device 10 may bepresent as shown in Table 1 below.

TABLE 1 Combinations of outputs and exemplary instructions Probabilityof the parent Al Quality of the outputting an erroneous semiconductorsubstrate output (output of the Instruction to the manufacturing (outputof the parent Al) evaluation device) apparatus Acceptable High Transportthe semiconductor substrate for further inspection. Acceptable Low Letthe semiconductor substrate proceed as a completed product. Notacceptable High Transport the semiconductor substrate for furtherinspection. Not acceptable Low Remove the semiconductor substrate fromthe production line.

The instruction generation unit 90 may be configured to generate aninstruction as shown in the right column of Table 1 depending on thecombination of the outputs from the parent AI and the evaluation device,for example. The instructions shown in Table 1 are merely exemplary andthe instruction generation unit 90 may be configured to generate otherinstructions with respect to the combinations of the outputs from theparent AI and the evaluation device.

The instruction generated by the instruction generation unit 90 may becommunicated to the manufacturing apparatus 40 via a communicationinterface (not shown) configured to communicate the instruction to themanufacturing apparatus 40. The manufacturing apparatus 40 may processthe semiconductor substrate according to the instruction received fromthe instruction generation unit 90.

The instruction generation unit 90 may be implemented by a generalpurpose computer having a configuration similar to that shown in FIG. 1for implementing the evaluation device. In some examples, theinstruction generation unit 90 may be implemented on the same generalpurpose computer on which the evaluation device 10 is implemented. Inother examples, the instruction generation unit 90 may be implemented ageneral purpose computer comprised in the semiconductor substratemanufacturing device 40. In yet other examples, the instructiongeneration unit 90 may be implemented as a device separate from both theevaluation device 10 and the semiconductor substrate manufacturingdevice 40.

Evaluation Process

FIG. 8 shows a flowchart of exemplary processing performed by theevaluation system. The processing of FIG. 8 may be started, for example,in response to a user input instructing the system to start theprocessing.

In step S10 of FIG. 8, the training of the parent AI may be performed asdescribed later in more detail with reference to FIG. 9.

After the parent AI is trained in step S10, the training of the NN 1020of the evaluation device 10 may be performed in step S20 as describedlater in more detail with reference to FIG. 10.

After the NN 1020 of the evaluation device 10 is trained in step S30 ofFIG. 8, a determination may be made as to whether or not to startevaluation of an output from the parent AI. It may be determined tostart the evaluation when a signal indicating start of manufacturing bythe manufacturing apparatus 40 is received at the parent AI and theevaluation device 10, for example. If it is determined to start theevaluation (YES at step S30), the processing may proceed to step S40. Ifnot (NO at step S30), the processing may perform the determination ofstep S30 again and wait until it is determined to start the evaluation.

In step S40 of FIG. 8, the parent AI may perform its operation toevaluate the quality of a substrate in an image captured by the camera50 as described later in more detail with reference to FIG. 11.

After step S40, in step S50 of FIG. 8, the evaluation device 10 mayevaluate the output from the parent AI as described later in more detailwith reference to FIG. 12.

After step S50, the evaluation device 10 may determine whether or not toend the evaluation process in step S55. For example, the evaluationdevice 10 may determine to end the evaluation process when theevaluation device 10 has no more image of a semiconductor substrate tobe evaluated and/or the evaluation device 10 receives a signal from themanufacturing apparatus 40 indicating that the manufacturing of thesemiconductor substrates have ended. Further, for example, theevaluation device 10 may determine not to end the evaluation processwhen the evaluation device 10 still has one or more images to beevaluated and/or the evaluation device 10 receives a signal from themanufacturing apparatus 40 indicating that the manufacturing of thesemiconductor substrates is still ongoing. When it is determined not toend the evaluation process (NO in step S55), the processing may returnto step S40. When it is determined to end the evaluation process (YES instep S55), the processing shown in FIG. 8 may end.

FIG. 9 shows a flowchart of exemplary processing for training the parentAI. FIG. 9 is an example of detailed processing of step S10 of FIG. 8.When step S10 of FIG. 8 is started, the processing shown in FIG. 9 maystart.

In step S100 of FIG. 9, training data may be prepared for training theparent AI by training data generation unit 706 of the parent AI. Forexample, the training data generation unit 706 may receive an image of asemiconductor substrate and information indicating the quality of thesemiconductor substrate via the receiving unit 700 of the parent AI andgenerate an element of the training data set, the element including acombination of the received image and the received quality information.The training data generation unit 706 may generate a specified number ofsuch elements of the training data set.

In step S102 of FIG. 9, the parent AI may be trained using the trainingdata generated in step S100, by the AI training unit 708 of the parentAI. For example, the AI training unit 708 of the parent AI may retrievedata structure of a CNN from the neural network DB 80 and train the CNNas the AI 7020 of the computation unit 702, by adjusting the weights ofthe convolutional layer(s) and the fully connected layer(s) as describedabove with reference to FIGS. 4 to 6.

After step S102, the AI training unit 708 may determine whether or notthe training is sufficient in step S104 of FIG. 9. In some examples, theAI training unit 708 may use, for the determination of step S104, testdata including one or more images of semiconductor substrates andinformation indicating quality of each of the semiconductor substratesin the images. The test data set may be prepared in a manner analogousto that for preparing the training data set in step S100. In someexamples, the AI training unit 708 may use a part of the training dataprepared in step S100 for training the parent AI in step S102 and theremaining part of the training data prepared in step S100 as the testdata for determining whether the training is sufficient in step S104. Inthe examples of using the test data in step S104, the AI training unit708 may input the images in the test data to the AI 7020 of thecomputation unit 702 and compare the outputs from the AI 7020 for theimages with the known quality of the semiconductor substrates incorresponding images. The AI training unit 708 may, for example,determine that the training is sufficient if a ratio of the number ofcorrect outputs from the AI 7020 over the total number of images in thetest data exceeds a predetermined threshold. Alternatively, for example,the AI training unit 708 may determine that the training is sufficientif the number of correct outputs from the AI 7020 exceeds apredetermined threshold. When it is determined that the training is notsufficient (NO in step S104), the processing may return to step S102.When it is determined that the training is sufficient (YES in stepS104), the processing may proceed to step S106.

In step S106, the AI training unit 708 may determine whether or notthere is (are) further subject(s) to be learnt by the parent AI. Forexample, in case the parent AI is desired to evaluate the quality ofmore than one kinds of semiconductor substrates and training datarelating to at least one of said more than one kinds of semiconductorsubstrates has not yet been generated in step S100, the AI training unit708 may determine that there is (are) further subject(s) to be learnt bythe parent AI. When it is determined that there is (are) furthersubject(s) to be learnt (YES in step S106), the processing may return tostep S100. Otherwise (NO in step S106), the processing shown in FIG. 9may end.

FIG. 10 shows a flowchart of exemplary processing for training a neuralnetwork of an evaluation device 10 comprised in the evaluation system.FIG. 10 is an example of detailed processing of step S20 of FIG. 8. Whenstep S20 of FIG. 8 is started, the processing shown in FIG. 10 maystart.

In step S200 of FIG. 10, the neural network training unit 106 of theevaluation device 10 may prepare training data for training the NN 1020of the evaluation device 10. For example, the training data may includeimages of semiconductor substrates and information indicating, for eachimage, a level of erroneousness of an output from the parent AI for saidimage.

In some examples, the neural network training unit 106 of the evaluationdevice 10 may prepare the training data in step S200 as follows. First,the neural network training unit 106 may receive images of manufacturedsemiconductor substrates and quality information indicating, for eachone of the received images, quality of a manufactured semiconductorsubstrate in said one of the received image. It is noted that thereceived image and the quality information may have the same format asthe training data for training the parent AI. Further, the neuralnetwork training unit 106 of the evaluation device 10 may provide one ofthe received images to the parent AI as an input, obtain an output fromthe parent AI in response to the provision of said one of the receivedimages, and compare the obtained output from the parent AI with thequality of the manufactured semiconductor substrate in the imageprovided to the parent AI. Based on the comparison, the neural networktraining unit 106 may generate an element of a training data set, theelement including a combination of an image of a semiconductor substrateand information indicating a level of erroneousness of an output fromthe parent AI for said image. The neural network training unit 106 maygenerate an element of the training data set for each of the receivedimage.

In step S202, the neural network training unit 106 may train the NN 1020of the evaluation device 10 using the training data prepared in stepS200. For example, the neural network training unit 106 may retrieve adata structure of a CNN from the neural network DB 80 and train the CNNas the NN 1020 in the determination unit 102, by adjusting the weightsof the convolutional layer(s) and the fully connected layer(s) asdescribed above with reference to FIGS. 4 to 6. Alternatively, forexample, the neural network training unit 106 may retrieve a datastructure of a neural network with more than three layers (e.g., asshown in FIG. 7A) and train the neural network using a deep learningtechnique involving an autoencoder as described above with reference toFIGS. 7A and 7B. In other examples, another type of neural network maybe used as the NN 1020. During training of the NN 1020, the informationindicating a level of erroneousness of an output from the parent AI foran image may be used as a supervisory signal.

After step 202, the neural network training unit 106 may determinewhether or not the training is sufficient. This determination may bemade in a manner analogous to that of step S102 of FIG. 9. When it isdetermined that the training is not sufficient (NO in step S204), theprocessing may return to step S202. When it is determined that thetraining is sufficient (YES in step S204), the processing may proceed tostep S206.

In step S206, the neural network training unit 106 may determine whetheror not there is (are) further subject(s) to be learnt by the NN 1020 ofthe evaluation device 10. This determination may be made in a manneranalogous to that of step S106 of FIG. 9. When it is determined thatthere is (are) further subject(s) to be learnt (YES in step S206), theprocessing may return to step S200. Otherwise (NO in step S206), theprocessing shown in FIG. 10 may end.

According to the processing of FIG. 10, the evaluation device 10 may beable to determine for which kind(s) of input images the parent AI islikely to provide an erroneous (or correct) output.

FIG. 11 shows a flowchart of exemplary processing performed by theparent AI. FIG. 11 is an example of detailed processing of step S40 ofFIG. 8. When step S40 of FIG. 8 is started, the processing shown in FIG.11 may start.

In step S400 of FIG. 11, the parent AI may receive input data, forexample, an image of a semiconductor substrate from the camera 50.

After step S400, the parent AI may perform computation using thereceived input data in step S402. For example, when a CNN is employed asthe AI 7020 of the parent AI, the parent AI may perform computation byinputting the received input data to the CNN and obtain an output fromthe CNN.

After step S402, the parent AI may output a result of the computation,for example, information indicating quality of the semiconductorsubstrate in step S404.

The processing of FIG. 11 may end after step S404.

FIG. 12 shows a flowchart of exemplary processing performed by theevaluation device. FIG. 12 is an example of detailed processing of stepS50 of FIG. 8. When step S50 of FIG. 8 is started, the processing shownin FIG. 12 may start.

In step S500 of FIG. 12, the receiving unit 100 of the evaluation device10 may receive input data, for example, an image of a semiconductorsubstrate from the camera 50.

After step S500, the determination unit 104 of the evaluation device 10may determine, in step S502, at least one value representative of aprobability of the parent AI outputting an erroneous output for theinput data received in step S500. For example, the determination unit104 may input the received image into the NN 1020 and obtain an outputfrom the NN 1020. The NN 1020 may be, for example, a CNN as describedabove with reference to FIGS. 4 to 6. Alternatively, for example, the NN1020 may be a neural network for deep learning as described above withreference to FIGS. 7A and 7B. Another type of neural network may be usedas the NN 1020 in other examples.

After step S502, the processing proceeds to step S504 and the outputunit 106 of the evaluation device 10 may output an output based on theat least one value representative of the probability determined in stepS502. The processing shown in FIG. 12 may end after step S504.

According to the processing of FIG. 12, information may be provided asto how likely the output of the parent AI is erroneous (or correct) fora particular input, which may contribute to improving overall accuracyof evaluation for a semiconductor substrate.

FIG. 13 shows a flowchart of exemplary processing for generating aninstruction to a semiconductor substrate manufacturing apparatus. Theprocessing shown in FIG. 13 may be started when the instructiongeneration unit 90 receives the outputs from the parent AI and theoutput unit 106 of the evaluation device 10.

In step S600 of FIG. 13, the instruction generation unit 90 may receivefrom the parent AI an output indicating quality of the substrate in aninput image.

Further, in step S602, the instruction generation unit 90 may receivefrom the evaluation device 10 an output based on the probability of theparent AI outputting an erroneous result for the input data.

After step S602, the instruction generation unit 90 may generate aninstruction to the manufacturing apparatus 40 in step S604. For example,one of the instructions shown in Table 1 above may be generateddepending on the combination of the outputs from the parent AI and theevaluation device received in steps S600 and S602.

After step S604, the instruction generation unit 90 may communicate thegenerated instruction to the manufacturing apparatus 40 in step S606.The processing shown in FIG. 13 may end after S606.

The manufacturing apparatus 40 may operate according to the instructioncommunicated from the instruction generation unit 90 in step S606 ofFIG. 13. This may contribute to improving the overall quality of thesemiconductor substrates manufactured by the manufacturing apparatus 40since the manufacturing apparatus 40 may process the manufacturedsemiconductor substrates in consideration of how likely the parent AImay erroneously determine the quality of each of the manufacturedsemiconductor substrates.

Variations

It should be appreciated by those skilled in the art that the exemplaryembodiments and their variations as described above with reference toFIGS. 1 to 13 are merely exemplary and other embodiments and variationsmay exist.

For example, in the exemplary embodiments and examples described above,both the parent AI and the evaluation device 10 take an image of asemiconductor substrate as an input to the AI 7020 and the NN 1020.However, in some examples, the parent AI and/or the evaluation device 10may take sensor information from one or more sensors (see e.g., sensors60 of FIG. 1) as a part of the input in addition to the image. The oneor more sensors may be one or more of: a temperature sensor, a humiditysensor, a brightness sensor; an atmospheric pressure sensor. In such acase, the predetermined format of input data for the parent AI mayrepresent data including an image and sensor information.

When the parent AI takes sensor information as a part of the input andin case the parent AI employs a CNN as described above with reference toFIGS. 4 to 6, for example, an additional fully connected layer may beinserted between the last fully connected layer and the output layer ofthe CNN (see e.g., FIG. 6), where the additional fully connected layermay have more number of nodes than the last fully connected layer. Theadditional fully connected layer may include, for example, nodescorresponding to the nodes of the last fully connected layer and one ormore additional nodes corresponding to the one or more sensors. Each ofthe one or more additional nodes may correspond to one of the one ormore sensors and may accept a value output from the correspondingsensor.

When the evaluation device 10 takes sensor information as a part ofinput and in case the NN 1020 has a configuration as shown in FIG. 7A, anode corresponding to each of available sensors may be provided in theinput layer of the NN 1020, in addition to the nodes corresponding topixel values in the input image, for example. In case the NN 1020 is aCNN as described above with reference to FIGS. 4 to 6, the sensorinformation may be incorporated into the CNN in a manner analogous tothat in case of the parent AI being a CNN as stated above. By takingsensor information as a part of input, the evaluation device 10 may beable to learn and determine under which environment (e.g. temperature,humidity, brightness, atmospheric pressure etc.) the parent AI couldoutput an erroneous output for which kind of images.

Further, for example, in case only the parent AI uses the sensorinformation as a part of input and the evaluation device 10 takes onlyimages without sensor information as inputs, the training of and thedetermination made by the evaluation device 10 can be performed usinginput data with a smaller size than the input data for the parent AI.

In any case, an input to the evaluation device 10 may include at least apart of an input to the parent AI.

Further, although the exemplary embodiments described above relate toevaluating the quality of a semiconductor substrate, the exemplaryembodiments and variations thereof can be applied also for evaluatingquality of any other kind of manufactured products. For theseapplications, images of manufactured products may be used for trainingthe parent AI and the NN 1020 of the evaluation device 10 in a manneranalogous to those for the exemplary embodiments as explained above.Then the parent AI may evaluate the quality of the manufactured productsfor which the parent AI has been trained and the evaluation device 10may evaluate the outputs of the parent AI in a manner analogous to thosefor the exemplary embodiments as explained above.

Moreover, the exemplary embodiments described above may be applied notonly for evaluating products manufactured by a manufacturing apparatusbut also for evaluating other kinds of products such as agriculturalproducts (e.g., fruits and vegetables). For example, the parent AI maybe trained to classify an agricultural product into two or more qualityclasses using an image of the agricultural product, in a manneranalogous to those for the exemplary embodiments as explained above.Further, the evaluation device 10 may be trained to determine theprobability of the parent AI outputting an erroneous output for aparticular image using images of the same kind of agricultural productin a manner analogous to those for the exemplary embodiments asexplained above.

Further application of the evaluation method in the exemplaryembodiments may include image recognition for medical diagnosis. Forexample, the parent AI may be trained to perform classification ofmedical images of at least an external or internal part of a human oranimal body, using a known machine learning method (including e.g., CNNsas described above). The evaluation device 10 may be trained todetermine for which medical images the parent AI is likely to outputerroneous outputs, using the medical images as inputs. In this case,medical images, outputs from the parent AI for the medical images andknown diagnosis for each image may be used for training the NN 1020 ofthe determination unit 102 included in the evaluation device 10. Themedical images may include, but may not be limited to, X-Ray images,computer tomography (CT) images, magnetic resonance imaging (MRI)images, ultrasonic images and images captured using an endoscope.

Moreover, the determination performed by the determination unit 102 ofthe evaluation device 10 in the exemplary embodiments may also beapplied in case of the parent AI dealing with a more general imagerecognition problem. For example, the parent AI may include a CNN (seee.g., FIGS. 4 to 6) trained to detect a particular object (e.g., a humanor an animal such as a cat, bird, dog etc.) within an image. In such anexample, the NN 1020 of the determination unit 1020 may be trained toevaluate how likely that the parent AI outputs an erroneous detectionresult for a particular image. For instance, images of different scenes,outputs of the parent AI for these images and information indicatingwhether or not the target object is present in each image can be usedfor training the NN 1020 of the determination unit 1020.

In relation to image recognition, for example, the parent AI may betrained to perform object recognition during driving of a vehicle. Forinstance, images captured during driving of a vehicle by one or moreimaging devices (e.g. cameras) provided on the vehicle may be used asinput data for the parent AI. The images may represent scenes in frontof the vehicle in case the images are captured by a camera provided atthe front of the vehicle, for example. Alternatively or additionally,the images may represent scenes at the back of the vehicle in case theimages are captured by a camera provided at the rear of the vehicle. Theparent AI may be trained to determine whether or not any targetobject(s) (e.g., a human, an animal, and/or any other object whichcollision with the vehicle is desired to be avoided) is/are present inthe scene represented by an input image. In this example, the parent AImay comprise a CNN as described above with reference to FIGS. 4 to 6 andthe CNN may be trained to detect one or more target objects using imagescaptured by one or more imaging devices provided on a vehicle.

In the exemplary application of object recognition during driving of avehicle, the parent AI may be more likely to provide erroneous resultsfor some scenes where the conditions of the road surface could bedifferent from usual conditions, such as scenes during night, sceneswith rain, snow and/or ice, etc. Further, in this example, thedetermination unit 102 of the evaluation device 10 may be trained todetermine how likely the parent AI will output an erroneous result ofobject recognition for images of which kind of scenes. The NN 1020 ofthe determination unit 102 may include a CNN (see e.g., FIGS. 4 to 6)and may be trained using images captured by one or more cameras providedon a vehicle, outputs of the parent AI for these images and informationindicating whether or not one or more target objects are present in eachimage. The information indicating whether or not one or more targetobjects are present in each image may be provided by a user. The outputsof the parent AI and the information indicating whether or not one ormore target objects are present in each image may be used for generatingsupervisory signals for training the NN 1020, indicating, for eachimage, a level of erroneous ness for an output from the parent AI forthat image. The NN 1020 of the determination unit 102 so trained candetermine, for example, that the parent AI may be more likely to outputan erroneous recognition result for an image of a scene including snowon the road.

Further, in the exemplary application of object recognition duringdriving of a vehicle, the NN 1020 may further take sensor information,as a part of input, from one or more sensors provided on the vehicle.For example, a temperature sensor provided on the vehicle to detectoutside air temperature. In such a case, the NN 1020 may be a CNN (seee.g., FIGS. 4 to 6) that includes an additional fully connected layerhaving a node corresponding to the sensor information from thetemperature sensor, as described above with regards to the use of sensorinformation in the examples of semiconductor substrate evaluation. Thesensor information from the temperature sensor may contribute toimproving accuracy of the determination made by the NN 1020 of thedetermination unit 102 since, for example, an outside air temperaturearound or below 0° C. may indicate the possible presence of snow and/orice on the road in which case the parent AI may be more likely to outputan erroneous recognition result. Additionally or alternatively to thetemperature sensor as stated above, the sensor information may beobtained from one or more of the following sensors provided on thevehicle: rain sensor, yaw rate sensor, inclination sensor, accelerationsensor, daylight sensor, radar sensor, IR (infrared) sensor.

In addition, the applications to the determination performed by thedetermination unit 102 of the evaluation device 10 of the exemplaryembodiments may not be limited to image analysis performed by the parentAI. Further applications may include, but may not be limited to,determination regarding an output from a parent AI that is trained toperform speech recognition or natural language processing, for example.

In case of speech recognition, the parent AI may comprise a CNN (seee.g., FIGS. 4 to 6) trained to recognize particular words in a speechusing audio data of utterances as training data, for instance. In theexample of speech recognition, the predetermined format of the inputdata for the parent AI may represent the audio data which is suitablefor use as inputs to a CNN. Further, in this example, the NN 1020 of thedetermination unit 102 included in the evaluation device 10 may betrained to determine at least one value representative of a probabilityof the parent AI outputting an erroneous output for particular audiodata. For instance, audio data of utterances, outputs from the parent AIfor the audio data and information indicating whether or not particularwords are included in the audio data can be used for training the NN1020 of the determination unit 102.

In case of natural language processing, the parent AI may comprise a CNN(see e.g., FIGS. 4 to 6) trained to perform classification of textswritten in a natural language, for example. In this example, thepredetermined format of the input data for the parent AI may representdata of the texts that may be pre-processed to be suitable for use asinputs to a CNN. Further, in this example, the NN 1020 of thedetermination unit 102 included in the evaluation device 10 may betrained to determine at least one value representative of a probabilityof the parent AI outputting an erroneous output for a particular text.For instance, data of texts, outputs from the parent AI for the data andinformation indicating the correct classification result for each textmay be used for training the NN 1020.

In yet another example, the parent AI may be trained to performestimation of a state of a driver of an automobile in a drivermonitoring device. In this example, the parent AI may take sensorinformation from various sensors provided on the automobile as inputsand determine the state of the driver. The determination unit 102 of theevaluation device 10 may determine how likely the parent AI will outputan erroneous output regarding the state of the driver for a particularinput. In this example, the NN 1020 of the determination unit 102 may betrained using the sensor information, outputs from the parent AI andinformation indicating the correct state of the driver for each elementof the sensor information.

In yet another example, the parent AI may be trained to diagnosevascular disease in a patient body such as in the brain and/or heart. Asinput data to the parent AI, physiological data such as heartbeat, bloodpressure, blood sugar level etc. of a human or animal subject(hereinafter, also referred to merely as a “subject”) may be used. Inthis example and variations thereof described herein, a “human or animalsubject” may be understood as a human or an animal subject to thediagnosis. In this example, the parent AI may be trained usingphysiological data as training data. For example, physiological data ofeach of a plurality of subjects may be collected and the parent AI maybe trained using the collected physiological data as the training dataand information indicating whether or not each of the plurality ofsubjects has vascular disease in the brain and/or heart as supervisorysignals.

In this exemplary application to diagnosis of vascular disease, thedetermination unit 102 of the evaluation device 10 may determine howlikely the parent AI will output an erroneous diagnosis for particularphysiological data. The NN 1020 of the determination unit 102 may betrained using physiological data of a plurality of subjects, outputs ofthe parent AI for the physiological data and information indicatingwhether each of the plurality of subjects has vascular disease in thebrain and/or heart. The information may be provided by each of theplurality of subjects himself/herself in case the subjects are human,for example. The outputs of the parent AI and the information indicatingwhether each of the plurality of subjects has vascular disease in thebrain and/or heart may be used for generating supervisory signals fortraining the NN 1020, indicating, for each of the plurality of subjects,a level of erroneousness for an output from the parent AI for thephysiological data of that subject.

Further in this exemplary application to diagnosis of vascular disease,the NN 1020 may take, in addition to the physiological data, an image ofa face of a human subject while the physiological data is obtained as apart of input data. The image may be captured by an imaging device whilethe physiological data is obtained from the human subject. The image maybe used for analyzing the facial expression of that human subject, e.g.,being angry, smiling, being nervous, being pale, etc. The NN 1020 maycomprise a CNN (see e.g., FIGS. 4 to 6) to process the image of the faceof the human subject whose physiological data is obtained. In such acase, the CNN may comprise an additional fully connected layer includingnodes for the image and one or more nodes corresponding to thephysiological data. Since it is known that the physiological data canlead to more accurate diagnosis if obtained while the human subject(e.g. patient) is in an ordinary state, using the image of the facialexpression of the human subject as a part of input data to the NN 1020may contribute to improving accuracy of the determination made by thedetermination unit 102 of the evaluation device 10. For example, thedetermination unit 102 with the NN 1020 trained using images of facialexpressions in addition to the physiological data may determine that theparent AI may be more likely to provide an erroneous diagnosis in casethe image shows an angry facial expression. In this example, the parentAI may also be configured to take an image of a face of the person as apart of input data, in a manner analogous to that with the NN 1020 asstated above.

The exemplary embodiments and variations thereof as described above mayinvolve a CNN (see e.g., FIGS. 4 to 6) as the parent AI. However, inother embodiments and variations, the parent AI may employ any one ofknown machine learning techniques.

As long as the NN 1020 of the evaluation device 10 is trained usinginput data having a format corresponding to the format of the input datafor the parent AI and using supervisory signals indicating for whichinput data the parent AI has output an erroneous output, the evaluationdevice 10 may be able to determine at least one value representative ofthe probability of the parent AI outputting an erroneous output for aparticular input.

Accordingly, in the exemplary embodiments and variations as describedherein, the evaluation device 10 may generally be considered as adetermination device for determining error of an output from a parent AIwhich is configured to: (i) receive input data in a predeterminedformat, (ii) perform computation using the input data, and (iii) providea result of the computation as the output.

Further, the following items provide some aspects of the presentdisclosure.

Item 1. A determination device (10) for determining error of an outputfrom a machine learning device (70) that is configured to: (i) receiveinput data in a predetermined format, (ii) perform computation using theinput data, and (iii) provide a result of the computation as the output,the determination device comprising:

-   -   a receiving unit (100) configured to receive data having a        format corresponding to the predetermined format;    -   a determination unit (102) configured to determine, using a        neural network (1020), at least one value representative of a        probability of the machine learning device (70) outputting an        erroneous output for input data corresponding to the received        data; and    -   an output unit (104) configured to output an output based on the        at least one value representative of the probability,    -   wherein the neural network (1020) has been trained using:        -   training data having the format corresponding to the            predetermined format; and        -   information indicating, for each element of the training            data, a level of erroneousness for an output from the            machine learning device (70) for input data corresponding to            the element of the training data.

Item 2. The determination device (10) according to item 1, wherein thetraining data used for training the neural network (1020) includes oneor more elements that are not included in training data used fortraining the machine learning device (70).

Item 3. The determination device (10) according to item 1 or 2, furthercomprising:

-   -   a neural network training unit (106) configured to train the        neural network (1020) using the training data having the format        corresponding to the predetermined format and the information        indicating, for each element of the training data, a level of        erroneousness for an output of the machine learning device (70)        for input data corresponding to the element of the training        data,    -   wherein the training of the neural network (1020) may be        performed according to deep learning technique, and    -   wherein the neural network training unit (106) may be further        configured to generate the information indicating the level of        erroneousness used for training the neural network (1020) by:    -   receiving the training data having the format corresponding to        the predetermined format and information indicating, for each        element of the training data, a correct output for input data        corresponding to the element of the training data;    -   providing an element of the training data to the machine        learning device (70) as an input;    -   obtaining an output from the machine learning device (70) in        response to the provision of the element of the training data;        and    -   comparing the obtained output from the machine learning device        (70) with the correct output for input data corresponding to the        element of the training data.

Item 4. A determination system comprising:

-   -   the determination device (10) according to any one of items 1 to        3; and    -   the machine learning device (70) configured to: (i) receive        input data in a predetermined format, (ii) perform computation        using the input data, and (iii) provide a result of the        computation as the output.

Item 5. A computer-implemented method for training a neural network(1020) to determine at least one value representative of a probabilityof a machine learning device (70) outputting an erroneous output forinput data having a predetermined format, the machine learning device(70) being configured to: (i) receive input data in a predeterminedformat, (ii) perform computation using the input data, and (iii) providea result of the computation as the output, the method comprising:

-   -   receiving data having a format corresponding to the        predetermined format and information indicating, for each        element of the received data, a level of erroneousness for an        output from the machine learning device for input data        corresponding to the element of the received data; and    -   training the neural network (1020) using the received data as        inputs to the neural network (1020) and the received information        as supervisory data, wherein the training may be according to        deep learning technique.

Item 6. A determination device (10) for determining error of an outputfrom a machine learning device (70), the determination devicecomprising:

-   -   a receiving unit (100) configured to receive an image of a        scene, the image being captured during driving of a vehicle by        an imaging device (50) provided on the vehicle;    -   a determination unit (102) configured to determine, using a        neural network (1020), at least one value representative of a        probability of a machine learning device (70) outputting an        erroneous output for the image of the scene, the machine        learning device (70) being configured to: (i) receive the image        of the scene, (ii) perform computation using the received image,        and (iii) output information indicating whether or not one or        more target objects are present in the scene based on a result        of the computation; and    -   an output unit (104) configured to output an output based on the        at least one value representative of the probability,    -   wherein the neural network (1020) has been trained using:        -   images of scenes captured by the imaging device provided on            the vehicle; and        -   information indicating, for each one of the images of the            scenes, a level of erroneousness for an output from the            machine learning device (70) for said one of the images of            the scenes.

Item 7. An object recognition system comprising:

-   -   the determination device (10) according to item 6; and    -   the machine learning device (70) configured to: (i) receive the        image of the scene, (ii) perform computation using the received        image, and (iii) output information indicating whether or not        one or more target objects are present in the scene based on a        result of the computation.

Item 8. A computer-implemented method for determining error of an outputfrom a machine learning device (70), the method comprising:

-   -   receiving an image of a scene, the image being captured during        driving of a vehicle by an imaging device (50) provided on the        vehicle;    -   determining, using a neural network (1020), at least one value        representative of a probability of a machine learning device        (70) outputting an erroneous output for the image of the scene,        the machine learning device (70) being configured to: (i)        receive the image of the scene, (ii) perform computation using        the received image, and (iii) output information indicating        whether or not one or more target objects are present in the        scene based on a result of the computation; and    -   outputting an output based on the at least one value        representative of the probability,    -   wherein the neural network (1020) has been trained using:        -   images of scenes captured by the imaging device provided on            the vehicle; and        -   information indicating, for each one of the images of the            scenes, a level of erroneousness for an output from the            machine learning device (70) for said one of the images of            the scenes.

Item 9. A determination device (10) for determining error of an outputfrom a machine learning device (70), the determination devicecomprising:

-   -   a receiving unit (100) configured to receive physiological data        of a human or animal subject;    -   a determination unit (102) configured to determine, using a        neural network (1020), at least one value representative of a        probability of a machine learning device (70) outputting an        erroneous output for the physiological data of the human or        animal subject, the machine learning device (70) being        configured to: (i) receive the physiological data of the human        or animal subject, (ii) perform computation using the received        physiological data, and (iii) output information indicating        whether or not the human or animal subject has vascular disease        in a brain and/or heart based on a result of the computation;        and    -   an output unit (104) configured to output an output based on the        at least one value representative of the probability,    -   wherein the neural network (1020) has been trained using:        -   physiological data of human or animal subjects; and        -   information indicating, for physiological data of each one            of the human or animal subjects, a level of erroneousness            for an output from the machine learning device (70) for the            physiological data of said one of the human or animal            subjects.

Item 10. A diagnostic system comprising:

-   -   the determination device (10) according to item 9; and    -   the machine learning device (70) configured to: (i) receive the        physiological data of the human or animal subject, (ii) perform        computation using the received physiological data, and (iii)        output information indicating whether or not the human or animal        subject has vascular disease in a brain and/or heart based on a        result of the computation.

Item 11. A computer-implemented method for determining error of anoutput from a machine learning device (70), the method comprising:

-   -   receiving physiological data of a human or animal subject;    -   determining, using a neural network (1020), at least one value        representative of a probability of a machine learning device        (70) outputting an erroneous output for the physiological data        of the human or animal subject, the machine learning device (70)        being configured to: (i) receive the physiological data of the        human or animal subject, (ii) perform computation using the        received physiological data, and (iii) output information        indicating whether or not the human or animal subject has        vascular disease in a brain and/or heart based on a result of        the computation; and    -   outputting an output based on the at least one value        representative of the probability,    -   wherein the neural network (1020) has been trained using:        -   physiological data of human or animal subjects; and        -   information indicating, for physiological data of each one            of the human or animal subjects, a level of erroneousness            for an output from the machine learning device (70) for the            physiological data of said one of the human or animal            subjects.

1. An evaluation device for evaluating quality of a semiconductorsubstrate manufactured by a semiconductor substrate manufacturingapparatus, the evaluation device comprising a processor configured witha program to perform operations comprising: operation as a receivingunit configured to receive an image of the semiconductor substrate, theimage being captured by an imaging device provided on the semiconductorsubstrate manufacturing apparatus; operation as a determination unitconfigured to determine, using a neural network, at least one valuerepresentative of a probability of a machine learning device outputtingan erroneous output for the image of the semiconductor substrate, themachine learning device being configured to: (i) receive the image ofthe semiconductor substrate, (ii) perform computation using the receivedimage, and (iii) output information indicating the quality of thesemiconductor substrate based on a result of the computation; andoperation as an output unit configured to output an output based on theat least one value representative of the probability, wherein the neuralnetwork is trained using: images of manufactured semiconductorsubstrates; and information indicating, for each one of the images ofthe manufactured semiconductor substrates, a level of erroneousness foran output from the machine learning device for the one of the images ofthe manufactured semiconductor substrates.
 2. The evaluation deviceaccording to claim 1, wherein the images used for training the neuralnetwork comprise one or more images that are not comprised in trainingdata used for training the machine learning device.
 3. The evaluationdevice according to claim 1, wherein the machine learning device isfurther configured to receive sensor information from one or moresensors provided in relation to the semiconductor substratemanufacturing apparatus and to perform the computation using the sensorinformation, and wherein the one or more sensors comprise one or moreof: a temperature sensor; a humidity sensor; a brightness sensor; and anatmospheric pressure sensor.
 4. The evaluation device according to claim3, wherein the neural network is further trained using the sensorinformation, and wherein the determination made by operation as thedetermination unit is based at least partially on the sensorinformation.
 5. The evaluation device according to claim 1, wherein theneural network is further trained using sensor information from one ormore sensors provided in relation to the semiconductor substratemanufacturing apparatus, the one or more sensors comprise one or moreof: a temperature sensor; a humidity sensor; a brightness sensor; and anatmospheric pressure sensor, and the determination made by operation asthe determination unit is based at least partially on the sensorinformation.
 6. The evaluation device according to claim 1, wherein theprocessor is configured with the program to perform operations furthercomprising: operation as a neural network training unit configured totrain the neural network using the images of the manufacturedsemiconductor substrates and the information indicating a level oferroneousness for an output from the machine learning device for each ofthe images of the manufactured semiconductor substrates, wherein thetraining of the neural network is performed according to a deep learningtechnique, and wherein the processor is configured with the program toperform operations such that operation as the neural network trainingunit is further configured to generate the information used for trainingthe neural network by: receiving the images of the manufacturedsemiconductor substrates and quality information indicating, for eachone of the received images, quality of a manufactured semiconductorsubstrate in the one of the received images; providing one of thereceived images to the machine learning device as an input; obtaining anoutput from the machine learning device in response to the provision ofthe one of the received images; and comparing the obtained output fromthe machine learning device with the quality of the manufacturedsemiconductor substrate in the image provided to the machine learningdevice, the quality of the manufactured semiconductor substrate beingindicated in the received quality information.
 7. An evaluation systemcomprising: the evaluation device according to claim 1; and the machinelearning device configured to (i) receive the image of the semiconductorsubstrate, (ii) perform computation using the received image, and (iii)output information indicating the quality of the semiconductor substratebased on a result of the computation.
 8. The evaluation system accordingto claim 7, further comprising: an instruction generation unitconfigured to generate an instruction to the semiconductor substratemanufacturing apparatus as to processing of the semiconductor substratebased on the output from the evaluation device and the output from themachine learning device for the image of the semiconductor substrate;and a communication interface configured to communicate the instructionto the semiconductor substrate manufacturing apparatus.
 9. Asemiconductor substrate manufacturing system comprising: the evaluationsystem according to claim 8; the semiconductor substrate manufacturingapparatus configured to manufacture the semiconductor substrate; and theimaging device provided on the semiconductor substrate manufacturingapparatus, wherein the semiconductor substrate manufacturing apparatusis further configured to: receive the instruction from the communicationinterface of the evaluation system; and process the semiconductorsubstrate according to the received instruction.
 10. An evaluationmethod for evaluating quality of a semiconductor substrate manufacturedby a semiconductor substrate manufacturing apparatus, the methodcomprising: receiving, by a processor, an image of the semiconductorsubstrate, the image being captured by an imaging device provided on thesemiconductor substrate manufacturing apparatus; determining, by theprocessor, using a neural network, at least one value representative ofa probability of a machine learning device outputting an erroneousoutput for the image of the semiconductor substrate, the machinelearning device being configured to: (i) receive the image of thesemiconductor substrate, (ii) perform computation using the receivedimage, (iii) and output information indicating the quality of thesemiconductor substrate based on a result of the computation; andoutputting, by the processor, an output based on the at least one valuerepresentative of the probability, wherein the neural network is trainedusing: images of manufactured semiconductor substrates; and informationindicating, for each one of the images of the manufactured semiconductorsubstrates, a level of erroneousness for an output from the machinelearning device for the one of the images of the manufacturedsemiconductor substrates.
 11. A method for training a neural network todetermine at least one value representative of a probability of amachine learning device outputting an erroneous output for an image of asemiconductor substrate, the machine learning device being configuredto: (i) receive the image of the semiconductor substrate, (ii) performcomputation using the received image, and (iii) output informationindicating quality of the semiconductor substrate based on a result ofthe computation, the method comprising: receiving images of manufacturedsemiconductor substrates and information indicating, for each one of theimages of the manufactured semiconductor substrates, a level oferroneousness for an output from the machine learning device for the oneof the images of the manufactured semiconductor substrates; and trainingthe neural network using the received images as inputs to the neuralnetwork and the received information as supervisory data, wherein thetraining is according to a deep learning technique.
 12. A non-transitorycomputer-readable storage medium storing a computer program comprisingcomputer-readable instructions that, when executed by a computer, causethe computer to perform the method according to claim
 10. 13. Anevaluation device for evaluating quality of a product manufactured by amanufacturing apparatus, the evaluation device comprising a processorconfigured with a program to perform operations comprising: operation asa receiving unit configured to receive an image of the product, theimage being captured by an imaging device provided on the manufacturingapparatus; operation as a determination unit configured to determine,using a neural network, at least one value representative of aprobability of a machine learning device outputting an erroneous outputfor the image of the product, the machine learning device beingconfigured to: (i) receive the image of the product, (ii) performcomputation using the received image, and (iii) output informationindicating the quality of the product based on a result of thecomputation; and operation as an output unit configured to output anoutput based on the at least one value representative of theprobability, wherein the neural network is trained using: images ofmanufactured products; and information indicating, for each one of theimages of the manufactured products, a level of erroneousness for anoutput from the machine learning device for the one of the images of themanufactured products.
 14. An evaluation method for evaluating qualityof a product manufactured by a manufacturing apparatus, the methodcomprising: receiving, by a processor, an image of the product, theimage being captured by an imaging device provided on the manufacturingapparatus; determining, by the processor, using a neural network, atleast one value representative of a probability of a machine learningdevice outputting an erroneous output for the image of the product, themachine learning device being configured to: (i) receive the image ofthe product, (ii) perform computation using the received image, and(iii) output information indicating the quality of the product based ona result of the computation; and outputting, by the processor, an outputbased on the at least one value representative of the probability,wherein the neural network is trained using: images of manufacturedproducts; and information indicating, for each one of the images of themanufactured products, a level of erroneousness for an output from themachine learning device for the one of the images of the manufacturedproducts.
 15. A determination device for determining error of an outputfrom a machine learning device that is configured to: (i) receive inputdata in a predetermined format, (ii) perform computation using the inputdata, and (iii) provide a result of the computation as the output, thedetermination device comprising a processor configured with a program toperform operations comprising: operation as a receiving unit configuredto receive data having a format corresponding to the predeterminedformat; operation as a determination unit configured to determine, usinga neural network, at least one value representative of a probability ofthe machine learning device outputting an erroneous output for inputdata corresponding to the received data; and operation as an output unitconfigured to output an output based on the at least one valuerepresentative of the probability, wherein the neural network is trainedusing: training data having the format corresponding to thepredetermined format; and information indicating, for each element ofthe training data, a level of erroneousness for an output from themachine learning device for input data corresponding to the element ofthe training data.
 16. A computer-implemented method for determiningerror of an output from a machine learning device that is configured to:(i) receive input data in a predetermined format, (ii) performcomputation using the input data, and (iii) provide a result of thecomputation as the output, the method comprising: receiving data havinga format corresponding to the predetermined format; determining, using aneural network, at least one value representative of a probability ofthe machine learning device outputting an erroneous output for inputdata corresponding to the received data; and outputting an output basedon the at least one value representative of the probability, wherein theneural network is trained using: training data having the formatcorresponding to the predetermined format; and information indicating,for each element of the training data, a level of erroneousness for anoutput from the machine learning device for input data corresponding tothe element of the training data.
 17. The evaluation device according toclaim 2, wherein the machine learning device is further configured toreceive sensor information from one or more sensors provided in relationto the semiconductor substrate manufacturing apparatus and to furtherperform the computation using the sensor information, and wherein theone or more sensors comprise one or more of: a temperature sensor; ahumidity sensor; a brightness sensor; and an atmospheric pressuresensor.
 18. The evaluation device according to claim 17, wherein theneural network is further trained using the sensor information, and thedetermination made by operation as the determination unit is based atleast partially on the sensor information.
 19. The evaluation deviceaccording to claim 2, wherein the neural network is further trainedusing sensor information from one or more sensors provided in relationto the semiconductor substrate manufacturing apparatus, the one or moresensors comprise one or more of: a temperature sensor; a humiditysensor; a brightness sensor; and an atmospheric pressure sensor, and thedetermination made by operation as the determination unit is based atleast partially on the sensor information.
 20. The evaluation deviceaccording to claim 2, wherein the processor is configured with theprogram to perform operations further comprising: operation as a neuralnetwork training unit configured to train the neural network using theimages of the manufactured semiconductor substrates and the informationindicating a level of erroneousness for an output from the machinelearning device for each of the images of the manufactured semiconductorsubstrates, wherein the training of the neural network is performedaccording to deep learning technique, and the processor is configuredwith the program to perform operations such that operation as the neuralnetwork training unit is further configured to generate the informationused for training the neural network by: receiving the images of themanufactured semiconductor substrates and quality informationindicating, for each one of the received images, quality of amanufactured semiconductor substrate in the one of the received images;providing one of the received images to the machine learning device asan input; obtaining an output from the machine learning device inresponse to the provision of the one of the received images; andcomparing the obtained output from the machine learning device with thequality of the manufactured semiconductor substrate in the imageprovided to the machine learning device, the quality of the manufacturedsemiconductor substrate being indicated in the received qualityinformation.